After two months performing abroad, Justin Timberlake finally reunited with his family in mid-September. “They were spotted at a pumpkin patch on Long Island, N.Y., but it wasn’t as happy as you’d expect,” a source exclusively tells In Touch of the singer, his wife, Jessica Biel, and sons Silas, 8, and Phineas, 4. “But Justin was lagging behind them and looked miserable.”

He wasn’t having the best week. The singer was in town to accept a deal to plead guilty to driving while impaired in Sag Harbor in June, for which he paid a $500 fine and will have to perform 25 to 40 hours of community service.

“The arrest really put a strain on his marriage,” says the source, adding that it was “very telling” that Justin announced a spate of new dates for his Forget Tomorrow World Tour in Europe for next year. “He’ll be on the road for another 10 months! They’ve been living separate lives and there’s no end in sight.”

Jessica has been holding down the fort while he’s on tour. “She’s been with the kids, but she’s also working on multiple projects,” the source says of the 42-year-old, who spent the summer working on her TV series, The Better Sister, in NYC. “It’s a lot, and her friends hate seeing her bogged down by all of this, and then having to deal with his humiliating arrest on top of everything else. They feel like Justin has been weighing her down for years, honestly.”

Justin Timberlake’s History of Bad Behavior

So far, she’s stood by his side. “He has a long history of getting himself into embarrassing situations, but they’ve always put up a united front in public,” says the source.

Justin, 43, has been accused of cheating on multiple occasions (which he’s denied) and has admitted to “getting plastered” and doing “my share of drugs” in the past.

“Behind the scenes, Jessica’s friends have talked to her about his behavior — they hate that so much of her life is about keeping Justin on track when he’s drinking and partying with friends,” says the source. “But she’s argued that they just don’t understand the Hollywood lifestyle.”

Still, it’s a lifestyle she’s tried hard to avoid. “She likes to keep the family private and moved them all to Montana a few years back, which helped remove Justin from temptations,” says the source. “They’ve tried everything to reconnect and are fighting to stay together for the sake of their family, but now that he’s traveling the world on tour, it’s going to get even harder. Friends are wondering if there are just too many issues to move past.”