Traveling by train between Italy and Slovenia is challenging.
There is no way!
But, fortunately, there are some alternatives. You can pick and choose, depending on the time and funds you can spend and your preferences, among the following:
1. Through Villach
You can use a train + bus combination that goes through Villach, Austria. This one takes about 6 hours and the trickiest part getting off at the right station, otherwise you may find yourself stranded on a tiny station for a couple of hours before the next train arrives. You can check the prices here. These tickets can be purchased only at your departure station.
2. Through Trieste
Option A: bus + train combination
There are a few buses running between Ljubljana and Trieste and vice versa that take between 1.5 and 2 hours and cost between 10 and 24 euros. You can check the schedule here. Take note that the Slovenian spelling for Trieste is TRST!
The fact that the bus station in Trieste is right next to the train station makes this option one of the most convenient budget options.
When you get there you just pick among the numerous trains departing to Venice almost every hour (schedule) with prices ranging from approximately 12 to 19 euros. Take note that the Italian spelling for Venice is VENEZIA!
Option B: train + bus + train (or train + tram + train)
This option goes through Villa Opicina, a small town very close to the Slovenian-Italian border, already in Italy. You can check the schedule of the train between Ljubljana and Villa Opicina here.
There is a bus or a historical tram that will get you between Villa Opicina and Trieste. The buses run pretty regularly and shouldn't cost more than a couple of euros, but interestingly their website is in Italian and not without its hiccups - I managed to get redirected to Chicago on Google maps.
The historical tram descends into Trieste almost like a funicular, offering great views of the Bay of Trieste and a unique experience with a historical feel. The trams run every 20 minutes between 7am and 8pm and take about 20 minutes. This option can be a wonderful treat, but beware, there is a 20 minute walk from the Villa Opicina train station to the tram station.
From Trieste you can then choose one of the trains I have already described in Option A.
3. Through Nova Gorica
If you are not in a rush and curious about other parts of Slovenia, there is an option that goes through Nova Gorica, a town on the Italian border, which was mainly built after WWII and shares its urban centre with its older sister across the border, Gorizia.
There are a few buses (1.5-2.5 hours, about 10 euros) and trains (3-4 hours, about 10 euros) from Ljubljana every day.
From the train station in Nova Gorica there is a bit of a walk (3.5km) crossing the invisible border to the train station in Gorizia, which you can avoid by using a taxi.
From Gorizia Centrale station you can easily hop on one of the trains heading to Venice (6 hours, 13-19 euros). You can find the schedule here.
4. Shuttle services
There are multiple providers of private services that offer transfers from Venice to Ljubljana and this is by far the quickest way. The prices vary, so it is best to check.