When humans vied with ChatGPT

Sure, AI is pulling off fantastic feats. But can it beat humans at creative tasks? Can it write songs that move you? DH ran a fun experiment pitting ChatGPT against humans. S R Ramakrishna brings you all the action.

Artificial intelligence (AI) became a huge talking point in 2023, thanks to ChatGPT and the host of new digital tools that followed in its wake. The collective wisdom among people in the arts was that AI was good for repetitive tasks, and could nudge you in the right direction when you ran into something like a writer’s block, but more creative work called for human emotions.

How good is ChatGPT, described as a chatbot, at writing lyrics? The Saturday Story pitted it against Neecia Majolly, well-known pianist, songwriter and singer, with a creative challenge. Is there a difference between how AI approaches the art of writing lyrics and how humans approach it? Neecia, who divides her time between India and Brunei, took up the challenge, and we had to come up with a theme.

Neecia suggested ‘Smart city’. I offered ‘Bengaluru traffic’. My younger colleagues in the DH features team wanted something more fun. They rallied for song titles by Michael Jackson, Taylor Swift and Coldplay. After animated discussions, we zeroed in on the title of a pop song, ‘Baby, calm down’, as the theme. The song, featuring Selena Gomez, was released in 2022, in the wake of the Covid lockdowns, and brought instant stardom to the Nigerian rapper Rema.

Neecia liked the idea. She gathered three musicians for the experiment. Three had to write their own songs, while one had to prompt ChatGPT to generate the fourth song. They got two weeks to write and make tunes. The concert, in the last week of December, was hosted by music school Odyssey. Located in Jayanagar, it is run by pianists Varsha Sastry and Akshay Govinde. About 15 people, from many walks of life, attended the concert.

Testing AI

But before we move to the concert action, a background. ‘Can you translate T S Eliot’s ‘The Waste Land’ into Kannada?’ was one of the first questions I typed into ChatGPT when I heard people raving about its AI capabilities.

A few months later, when I asked it to write a song in the style of Purandaradasa, it came up with a Kannada composition that used real words, but the versification was awful. Purandaradasa (16th century), revered as the father of Karnatik music, is a model of elegant simplicity; this AI-generated song was utter nonsense. Judging by the quality of these lyrics, you would conclude AI has a long way to go, and is taking on foolhardy tasks at the moment. But ChatGPT’s answer to the Eliot question, by contrast, was humble, almost human:

“Translating a complex work like T S Eliot’s ‘The Waste Land’ into another language requires a significant amount of time and expertise. As an AI language model, I can certainly help with translations, but it would be quite challenging to provide a complete translation of ‘The Waste Land’ into Kannada within the constraints of this chat. Translating literary works involves capturing not just the words, but also the nuances, metaphors, and cultural references present in the original text. It requires a deep understanding of both languages and their respective literary traditions. I recommend consulting a professional translator or Kannada literary expert for an accurate and nuanced translation of ‘The Waste Land’ into Kannada.”

The ability to write good poetry is regarded as one of the most sophisticated achievements of the human consciousness. For a long time, it was beyond our imagination to think a machine could do it. In the early ’90s, we heard about computer programs that could write poems and play chess. Three decades later, the advent of generative AI tools is springing many new surprises — AI bots can now write reports, essays, poems and songs.

ChatGPT, launched in November 2022, can scan the Net and come up with text compositions that would take humans hours, if not days, to write. It can instantaneously summarise literature available online on any subject. Similar bots are specialising in composing music, creating pictures, animating your doodles, and doing voice-overs. AI is pulling off many creative activities till recently regarded as quintessentially human.

Pop inspiration

The lyrics of the wildly popular ‘Baby calm down’ go: “Baby, calm down, calm down/Girl, this your body you put my heart for lockdown/For lockdown, oh, lockdown/Girl, you sweet like Fanta, Fanta.” Barkha Kumari, from the DH team, who emceed the event, was disappointed no one in the room particularly liked the song. It’s so dancey, she said, in its defence.

The concert featured four songs sung by three musicians. Neecia sang two — one she said she had written, and another by Prakash Savariappa. Her students Christine Azel John and Megha Rajesh Bhat, both 15, also performed the songs they wrote.

At the end of the 15-minute performance, Barkha revealed to the audience that one of the four songs was generated by ChatGPT, and asked if they could guess which one it was. Six of 15 got it right. Barkha then distributed the lyrics, and asked the audience to vote again. The singers obliged with another round of singing. Eight in the audience got it right this time. That was an interesting find: more than half the audience could guess which song was generated by AI, and which ones were written by humans.

Not everyone found it easy though. “Going by the lyrics, it is impossible to distinguish human lyrics from AI lyrics,” said Rajesh Bhat, an IT professional, who had volunteered to listen in.

We then asked the ones who had succeeded in detecting ChatGPT’s work how they had done it. The AI song begins with the lines, ‘Hey there little beach bum in your tiny shades/Sandcastles falling like political escapades…’.

Kavya Bhat, who is pursuing physics, computer science, and music theatre, first guessed Prakash’s song ‘When I sit to write a song…’ was AI-generated because of the expression ‘just tree,’ which she found “too bizarre”. “I felt a human could not write that. But after reading all the lyrics, I felt it wasn’t literal. I switched to (ChatGPT’s) ‘Hey there little beach bum…’ because it was too literal,” she said.

Mathangi Parthasarathi mistook Prakash’s lyrics for AI generated work because the “words didn’t flow well”, jumping from ‘just tree’ (which many in the audience found awkward) to ‘eternal monkeys’ and ‘monsters’.

Akshay, a pianist, couldn’t spot the machine among the humans. He voted and stuck to Christine’s song. “I chose it because I’ve gone down the ChatGPT rabbit hole for a year now and it can throw up surprisingly deep stuff,” he explained.

On a personal note, I got it wrong, too. I thought Megha’s song, a simple lullaby, was generated by ChatGPT as I heard vague echoes of the popular George Gershwin song ‘Summertime’ (1935) in it. My earlier interactions with the chatbot had shown how its creations were aggregated from multiple sources. “The little girl couldn’t get to bed/’Cause of the wildest dreams that ran through her head,” began Megha’s song. ‘Summertime’ expresses a similar sentiment: “...hush little baby, don’t you cry/One of these mornings you gonna rise up singin’”.

Writing humour is not easy, and the ChatGPT song had some strikingly funny phrases. I was particularly struck by ‘sandcastles falling like political escapades’, ‘sunscreen shutdown’ and ‘giggles-only countdown’.

It was Neecia who had given the prompts to ChatGPT. She explained how the process went: “I began with the prompt ‘Baby calm down song’. It yielded a song about the 9 to 5 rat race. It didn’t make sense to me as it did not have the structure of a song. Then I gave more prompts like ‘humorous’, and ‘political’.”

Neecia, who had never used a generative AI tool before, found formal and stylistic problems in what ChatGPT was writing. “I felt it was taking stuff from wherever and banding them together. I did six or seven iterations before I arrived at what I performed,” she said.

The human lyricists had invested their personal emotions into what they wrote, Neecia observed. Christine saw in the title an opportunity to write about the Gaza conflict. In her song, the lovers are separated by the raging violence: “Longing to see you, but left with only memories of our time/So baby, please calm down, there’ll always be tomorrow.”

Prakash is reading up on Buddhism, and his interest was reflected in his song. “All I ever want is — be free/I want to see… just tree/Baby, calm down, won’t you?” The cryptic ‘just tree’, he explained, was an idiom to say ‘keep things simple’, and also referred to the ability to see something as it exists in the moment. 

Neecia suspects AI is playing a big part in much of today’s pop music already: “Some of the lyrics are just terrible, they are repetitive all over. Everything is so overproduced. I don’t hear the humanness in the voice.” She says she often finds her teen students relating better to songs from many decades ago.

Everyday AI

Many in the audience were familiar with AI tools. PR professional Alex Mathew had once used it to generate pictures and edit photos. “Everybody on Instagram is using ChatGPT to write video scripts,” he said. When he tried it for scripting, he wasn’t happy. “It gives US-centric and not India-centric results,” he said.

Serene John, IB teacher of biology and theory of knowledge, found ChatGPT’s output ‘superficial’. “It just plays around with words. There is no depth,” she said.

When she was teaching philosophy, she gave the same problem to her students and ChatGPT. “My students were definitely better,” she said. But she acknowledged that ChatGPT gives pointers for essays that human intelligence can build on. A young student in the audience said she uses AI to generate ideas to paint and write poetry. 

IT professional Rajesh Bhat said AI fails on the emotion part but it is efficient in tasks like creating an itinerary for a week-long tour of Bengaluru.

Varsha Sastry used ChatGPT to create a write-up for an event for her music school. “The results were useless, though a bit funny. It doesn’t have a creative element. It only knows where to pull things from,” she said

Back in office

A week later, I asked ChatGPT, “How good are you at writing lyrics?”. It had a candid answer: “I can certainly help generate lyrics based on specific themes, styles, or emotions you’re looking for. I can provide creative and original content, but keep in mind that I don’t have personal experiences or emotions, so my lyrics won’t have the same depth as those written by human songwriters.”

It is early days for AI. But what would you say about its present status? Is it silly and reckless? Or sincere and almost human?

Here are a few lines from the ChatGPT-generated song, with prompts by Neecia Majolly:

Hey there little beach bum in your tiny shades,
Sandcastles falling like political escapades,
But baby, don’t let the tide bring you down,
Baby, calm down, it’s just vacation in this town.

Calm down baby, it’s not a sunscreen shutdown
Just waves crashing, not a bedtime noun
No need for tears, let’s not wear a vacation frown,
Oh baby, calm down, it’s a giggles-only countdown

Like this story? Email: [e-mail]