Done with forethought and strategy it can be a big time positive. But most people don't use forethought and strategy.
For example, in my early career, making less than I do now, I knew I wanted to save money to buy a home. I knew a 401k loan was an option on my plan. Instead of contributing only up to the employer match, paying taxes on the rest, and putting that money in a savings account, I maxed my contributions to build my 401k with the intent of eventually borrowing from it to buy a home. In that frame I had 2 scenarios laid out before me.
#1 Contribute 5% of my salary and pay income taxes on the rest, saving all other expendable income for the house.
#2 Contribute the max, not paying tax on it, and letting those contributions grow UNTIL I was ready to buy my home.
Option 2 gave me more money available to buy my home because I never paid income taxes on the pre-tax 401k contributions. My mortgage balance was significantly smaller, saving a great deal of interest paid to the bank. You get better rates when you can decrease the loan term length and your loan-to-value ratio is better. Does your 401k lose the potential growth? Yes, it does, and if you look at it with that tunnel vision it looks bad. But when you consider the entire picture it can be very financially advantageous to you. In the end, my 401k balance is much larger today than if I had chosen the alternative of only contributing to max the match. Once you get to later career and are likely maxing contribution limits you can't get back those early years where you possibly passed up the opportunity for maxing contributions in lieu of saving for a house etc. Why not do both?
Some cautions:
#1 if your 401k has any Roth balance and you cannot differentiate between pre-tax and Roth when initiating the loan then you will be additionally disadvantaged and should consider not taking the loan. For me, my employer did not offer Roth at the time so my entire 401k was pre-tax. They now offer Roth (I'm heavily into Roth now) and they do not differentiate. So if my 401k is 75% Roth and 25% pretax and I initiate a loan, 75% of the loan balance will come from my Roth. Reducing income potential for your Roth is counterproductive because you didn't have the tax advantage on the front end to offset the negative. If I were just starting my career I cannot say I would do it again simply for the fact that the Roth option on my 401k complicates things; it wasn't an option for me then. Every 401k plan is different and the possible benefits depend on the details of your exact 401k plan.
#2 Never borrow more than your emergency fund plus any other backup plans you may have. If you seperate from your job (voluntarily or involuntarily) you have a limited amount of time to repay the loan or face taxes, fees, & penalties associated with an unqualified distribution from your 401k. For me, I had a full year emergency fund plus I had a safety net in my parents who are always willing to give short term loans if needed.