One disembarking passenger with a cockapoo and a pair of binoculars greets someone in the queue. “We saw a fin whale,” I hear him say. “Keep your eyes peeled.”

This is exciting information. The Scillonian ferry is reputedly a great platform for spotting cetaceans and it’s a perfect day for it – the sea is calm and visibility is superb. From the deck, the promontory that is Land’s End actually seems dramatic and special, in a way that it doesn’t from dry land. There are several people armed with scopes and sights who are clearly experienced and observant. The only thing lacking is the animals. Not a single dolphin makes an appearance, never mind the others that make regular summertime splashes: humpbacks, minke, sunfish, basking sharks and, increasingly, bluefin tuna.

Arriving in Scilly by ship is worth the crossing: wild headlands, savage rocks, white sand beaches, sudden strips of transcendentally turquoise ocean interspersed with the bronzed pawprints of kelp. Of course, it can be thick mist and squalls, but we’re in luck, the islands are doing their best Caribbean impersonation. Hugh Town, the capital of St Mary’s, is built on the narrow isthmus between two rocky outcrops. It’s a quirky, independent town with the kind of traffic levels our grandparents would recognise.

Up the hill, from the terrace of the Star Castle Hotel, we can see all the islands spread out around us, and handily there’s a lady with a friendly labrador who gives us a pithy summary of each. St Martin’s: “Beach life.” Tresco: “The royals love it.” St Agnes: “Arty.” Bryher: “Wild and natural.”

Bryher is our big wildlife destination because the plan is to rent kayaks there and paddle to the uninhabited Samson island, which is a protected wildlife area. I’m banking on Samson for wildlife now that the whales didn’t show up, but first we’re going to explore St Agnes with Vickie from the Isles of Scilly Wildlife Trust.

After a short ferry ride from St Mary’s quay, we stroll around St Agnes and across a short sand spit, a tombolo, to its neighbour, Gugh. Vickie leads us up a heather-covered hillside next to an impressive stack of pink granite boulders. “St Agnes and Gugh used to have a rat problem,” she tells us. “There were an estimated 4,000 that had destroyed the breeding populations of both Manx shearwaters and storm petrels. We’re pretty sure we’ve eradicated them now and the bird populations are rising fast.”

She leans over a small burrow under a lichen-crusted rock, and sniffs. “Yes, that’s storm petrel – they have a distinctive aroma.” Using her phone, she plays a series of cackles and squeaks down the hole. No response.

I ask Vickie about the archipelago’s endemic species. The Scilly bee? “Hasn’t been seen for many years.” She chuckles. “What makes the islands special is often what we don’t have. There are no magpies or buzzards, no foxes or grey squirrels. Those absences are important.”

What they don’t have in terms of fauna, they certainly make up for in flora. The lanes and paths of St Agnes are a ravishing spectacle: agapanthus and honeysuckle, huge spires of echium and smooth succulent aeoniums from the Canary Islands. In this frost-free environment, all kinds of subtropical plants thrive, making the islands quite unlike anywhere else in the British Isles. Dotted among all this fecundity are artists’ studios, galleries, a pub and a community hall where there’s a wonderful display of shipwreck souvenirs: East India Company musket parts, skeins of silk, porcelain and perfume.

Back on St Mary’s, we swim and spot a seal. But if we imagine our luck is changing, it’s not. Next morning we are down on the quayside, bright and early for the boat to Bryher. “It just left,” says the ticket seller. “We did post the change last night. Very low tide. Had to leave 15 minutes early.”

“When is the next one?”

“There isn’t one.”

The islands, I should have known, are run by the tides. Be warned.

Without any time to think, we jump on the Tresco boat. A fellow passenger offers sympathy. “Last week we missed the boat from St Martin’s and had to spend the night there. It was great.”

I relax. She is right. The best travel adventures come unplanned.

The low tide means we land at Crow Point, the southern tip of Tresco. “Last return boat at five!” shouts the boatman. We wander towards a belt of trees, the windbreak for Tresco Abbey Garden. The eccentric owner of the islands during the mid-19th century, Augustus Smith, was determined to make the ruins of a Benedictine abbey into the finest garden in Britain. Having planted a protective belt of Monterey pine, his gardeners introduced a bewildering array of specimen plants from South Africa, Latin America and Asia: dandelions that are three and a half metres tall, cabbage trees and stately palms. Just to complete the surreal aspect, Smith added red squirrels and golden pheasants, which now thrive.

Now comes the moment, the adventure decision moment. I examine the map of the island and point to the north end: “It looks wilder up there, and there’s a sea cave marked.”

We set off. Tresco has two settlements: New Grimsby and Old Grimsby, both clutches of attractive stone cottages decked with flowers. Beyond is a craggy coast that encloses a barren moorland dotted with bronze age cairns and long-abandoned forts. At the north-eastern tip we discover a cave high on the cliffside. Now the low tide is in our favour. We clamber inside, using our phone torches. A ramp of boulders takes us down into the bowels of the Earth, and to our surprise, where the water begins, there is a boat, with a paddle. Behind it the water glitters, echoing away into absolute darkness.

We climb in and set off. Behind us and above, the white disc of the cave entrance disappears behind a rock wall. The sound of water is amplified. After about 50 metres we come to a shingle beach. “How cool is that?” says Maddy. “An underground beach.”

We jump out and set off deeper into the cave, which gets narrower and finally ends. On a rock, someone has placed a playing card: the joker.

Later that day, having made sure we do not miss the last boat back, we meet Rafe, who runs boat trips for the Star Castle Hotel. He takes pity on us for our lack of wildlife. “Come out on my boat tomorrow morning and we’ll see what we can find.”

Rafe is as good as his word. We tour St Martin’s then head out for the uninhabited Eastern Isles. Rafe points out kittiwakes and fulmars, but finally we round the rock called Innisvouls and suddenly there are seals everywhere, perched on rocks like altar stones from the bronze age. “They lie down and the tide drops,” says Rafe. “These are Atlantic greys and the males can be huge – up to 300kg.”

Impressive as the seals are, the islands are better known for birds, regularly turning up rarities. While we are there, I later discover, more acute observers have spotted American cliff swallows that have drifted across the Atlantic, various unusual shearwater species and a south polar skua.

Next day is our return to Penzance, and it’s perfect whale-watching weather. People are poised with binoculars and scopes, sharing tales of awesome previous sightings: the leaping humpbacks, the wild feeding frenzies of tuna, and the wake-riding dolphins. Nothing shows up. I complain, just a little, about our lack of wildlife luck. Maddy is playing with a pair of terriers. “The thing with Wilf was he was always content with whatever happened,” she says. I lounge back on the wooden bench on the port side, enjoying the wind, sun and sound of the sea. I’m channelling the spirit of Wilf. Be happy. Whatever. It’s a lovely voyage anyway. And that’s how I missed the sighting of the fin whale off the starboard side.

The Star Castle Hotel on St Mary’s has double rooms from £249 half-board off-season to £448 in summer; singles from £146 to £244. Woodstock Ark is a secluded cabin in Cornwall, handy for departure from Penzance South Pier (sleeps two from £133 a night). The Scillonian ferry runs March to early November from £75pp. Kayak hire on Bryher £45 for a half day, from Hut 62. For further wildlife information check out the ios-wildlifetrust.org.uk