Don't worry so much about what people think of you. Rather think about what God thinks of you. You cannot satisfy both parties. We are called to follow Jesus without compromising the Truth. Jesus said to His disciples: "Follow Me." (Matt 4:19) With His help, you can do things even better and prune away unfruitful, "dry branches" on your tree of life. It has to do with sanctification.

Jesus says:

"Every branch in Me that does not bear fruit He takes away; and every branch that bears fruit He prunes, that it may bear more fruit." (John 15:2) Jesus' desire for all believers: "That you bear much fruit."

John 15:8

In other words: Our Christian life should not stand still but grow day by day through the power given by the Holy Spirit. The fruits come out of a life fellowship with Christ. 

Jesus says:

"For without Me you can do nothing."

John 15:5

Here, it's not about "pulling yourself together" to grow, but rather just being a branch on the trunk, Jesus, and receiving from the power of the tree.

There are many types of fruit in the Christian life. Paul mentions nine fruits of the Holy Spirit (Gal 5:22). In addition, praise is mentioned: "the fruit of our lips, giving thanks to His name." (Hebrews 13:15)

And finally: Progress of the gospel is a great and important fruit. Paul says that "the gospel … is bringing forth fruit and growing" (Colossians 1:6). The salvation of many people must surely be the most important of all fruits! That is what we need more of!