The Lady in the Van may be a great comic character, but she is one of the unlikelier Britons ever to earn a blue plaque in her honour. Yet one now adorns the façade of 23 Gloucester Crescent, the London address that she besieged for fifteen years. Alan Bennett – her landlord of sorts – published the original story in the London Review of Books in 1989. Nicholas Hytner (who directs the new film) directed a stage version with Maggie Smith in 1999, when she was in her mid-sixties. It's a wonder they didn't film it then. But time has only enhanced Smith's portrayal of this quotable eccentric. The film was actually shot at 23 Gloucester Crescent; Bennett still owns it, though he doesn't live there. Maggie Smith, in an interview, said of the filming: 'It was haunting, and I also felt a bit guilty.'

Bennett – a veteran of Beyond the Fringe – was already famous when Miss Shepherd steered her putrid van into Gloucester Crescent, Camden Town, and applied the brake with the force of Excalibur. Gloucester Crescent was home to fashionable writers, editors, bohemians, directors, musicians. (A century earlier Charles Dickens consigned his abandoned wife to the Crescent – Catherine Dickens, another outcast.)

At first, Miss Shepherd was content to stay parked on the street. Bennett (who wrote the script) has fun mocking the pretensions of the Camdenites, who observe the newcomer with a kind of appalled fascination. In 1977, on the day of the Golden Jubilee, Miss Shepherd's was the only 'domicile' festooned with royal paraphernalia. One Christmas, a local gingerly presented her with a bowl of crème brûlée, of all things. The look on Maggie Smith's face as she accepts the dessert is worth the price of the ticket.

Finally, Miss Shepherd, after various assaults and indignities, persuaded the long-suffering Bennett to allow her to park her van in his driveway, directly below his study and his desk. She stayed for fifteen years – amid the most appalling conditions. The interior of the van was 'a midden of old clothes, plastic bags and half-eaten food'. Sanitation was limited to plastic bags which she would hurl onto the driveway from her window for Bennett to step on or remove. Not for her 'the pathetic, saving neatness of the poor', to borrow Susan Cheever's phrase. The stench around the van was indescribable. One postman remarked, 'The smell sometimes knocks you back a bit.'

Miss Shepherd was a master of indirection and avoidance, but gradually Bennett gleaned some facts about her circumstances, and these were supplemented after her death. A talented pianist, Miss Shepherd studied with Cortot in Paris, then drove ambulances during World War II and narrowly escaped death when a bomb exploded nearby. There may have been a fiancé who died during the war, but her brother doubted this rumour. Later, she became a novice in a convent at the end of Gloucester Crescent, until the nuns found her too rebellious. A crisis followed, and her brother had her committed to an asylum.

The film begins with a blackened screen and the sound of a terrible road crash. Only after her death did Bennett learn of the incident that changed her life forever. When a young motorist ploughed into her car and was killed, Miss Shepherd wrongly blamed herself and fled, becoming a kind of renegade, forever penitent and changing her name.

Like many itinerants, Miss Shepherd was not deceived by the kindness of strangers. Prayerful and withdrawn, she ignored social decorum and did not suffer middle-class patronisers. Bennett, in the story, writes of her 'querulous and often resentful demeanour'. She was a ruthless egoist – demanding, ungrateful, selfish. Michelet said, 'He who knows how to be poor knows something.' Miss Shepherd clearly knew something about the limits of compassion and the need for perpetual vigilance. Her curtness to strangers was legendary. 'But I only asked if you were all right,' wailed one shaken passer-by.

Maggie Smith as Miss Shepherd in The Lady in the Van (Sony Pictures)

Bennett's reasons for tolerating this epic and odoriferous visit are interesting. He does not depict himself as an instinctive Samaritan: he is too disengaged, too ironic, too fastidious. In many ways this is a lacerating self-portrait. After Miss Shepherd uses his toilet, Bennett furiously scrubs and disinfects the toilet bowl. Thereafter he discourages invasions of his house – 'it was here, on the threshold of the toilet, that my charity stopped short'. Bennett's mother ('Mam', beautifully played by Gwen Taylor), dementing and ultimately institutionalised during Miss Shepherd's long reign, is not mentioned in the original story, but here she is a presence: loving, bewildered, and slightly embarrassing.

As in the play, Hytner and Bennett depict the latter's ambivalence by splitting his personality. There is 'the self who does the living' and 'the self who does the writing'. Often they appear on screen together, squabbling like an odd couple (technically, the doubling is cleverly done). The writing self, never leaving his desk, wryly observes his alter ego's halting attempts to deal with Miss Shepherd. Always there is the lure of good copy, the posthumous book/play/film. Germane here is a line from Borges that Bennett has quoted before: 'All the books he ever published filled him with a complex feeling of repentance.'

At first we suspect that The Lady in the Van will be another of those light-hearted British comedies with an impeccable pedigree and a soggy middle, but the film transcends celebrity and niceness in its exploration of different and darker territory. Charity – its rituals, its absurdities, its mutual humiliations – is the theme of the film. Bennett, at one point, berates a social worker and denies being Miss Shepherd's carer; denies indeed the reality or possibility of 'care', especially in London.

The seasoned cast relishes Bennett's masterly lines. Everyone is in it. At times we're not sure which drama series we're in, which century, which class. Is it Cranford, Downton Abbey, The History Boys? Frances de la Tour is particularly good as Mrs Vaughan Williams, constantly amused by the Camden farce.

Alex Jennings is outstanding as Bennett: he even perfects that unique and throttled accent. Maggie Smith – perhaps the most mannered actress since Gloria Swanson or Marilyn Monroe – entertains us with vowels and tics and glares and postures that are by now utterly familiar. That extraordinary Audenesque face has never been so expressive, so exposed. But there is a new dimension to this performance, which must be the best thing she has done in years. Miss Shepherd's thwarted musicality (not accentuated in the story) is important here, and a couple of scenes linger in the memory: in the first one, she tells Bennett about her ordeal at the convent and why she stopped playing.

Alex Jennings as Alan Bennett in The Lady in the Van (Sony Pictures)

Not all of the new material or characters works. The film introduces a sub-plot in which Jim Broadbent, playing a retired policeman who knows about the collision, blackmails Miss Shepherd in later life. Maybe this was a way of enlisting Broadbent, another Bennett stalwart, but these scenes feel gauche.

As with all natural comedians, Miss Shepherd's sentences were perfect – if we are to believe Alan Bennett, whose film is, after all, subtitled 'a mostly true story'. (He should have added the 'possibly' with which Miss Shepherd qualified many of her statements.) Here are some examples: 'What you don't understand is that I am insured. I am insured in heaven.' 'I have a testimonial for a Clean Room, awarded some years ago, and my aunt, herself spotless, said I was the cleanest of my mother's children, particularly in the unseen places.' One day Miss Shepherd asked Bennett: 'When you were in Yugoslavia, did you come across the Virgin Mary?' (This reminded me of the time I overheard a pair of septuagenarians, one of whom said to the other, whose head was under a car bonnet at the time: 'When were you last in the Holy Land?' I used the line in one of my poems. Material like that is too good to ignore, and Miss Shepherd gave her landlord an endless supply.)

Miss Shepherd's health began to fail in her seventies. Eventually, she agreed to go to a day centre. There, finally, she was bathed, fed, placed in clean sheets, cared for. In the cafeteria she found a piano and played Chopin for the first time in many years. In the film, when she returns to Gloucester Crescent there is a belated physical gesture between Miss Shepherd and Alan Bennett ('Hold my hand, Mr Bennett. It's clean'). But in the story the gesture, fittingly, is purely verbal. 'I'm sorry the van is in such a state. I haven't been able to do any spring cleaning.' There it all is: pride; awkwardness, thwarted communion; a kind of regard.

The next day, a social worker (not Bennett, to his regret) found Miss Shepherd dead in the van. That day Bennett wrote in the diary that he mines in the story (and virtually everything he writes): 'I am filled with remorse for my harsh conduct towards her, though I know at the same time that it was not harsh. But still I never quite believe or chose to believe she was as ill as she was, and I regret too all the questions I never asked her. Not that she would have answered them. I have a strong impulse to stand at the gate and tell anyone who passes.'

Lady in the Van (PG-13), directed by Nicholas Hytner, distributed in Australia by Sony Pictures. In cinemas March 3.

Arts Update is generously supported by The Ian Potter Foundation.