Musical Instruments Playing Poetry

• The idea originated from a conversation with ritar.
• On May 18, 2024, while talking to tskokmt, a specific vision emerged.

  • When creating the prototype of “OBJECT,” a musical instrument that plays visuals we are developing with @taito_hasegawa, we paid tribute to past outstanding visuals by sampling patterns devised by Dutch graphic designer Karel Martens, allowing our generation to interact and play with them#modulepoeticcode pic.twitter.com/SFJ01oRtex

  • @(TSKOKMT) [March 24, 2024](https://twitter.com/TSKOKMT/status/1771910114527826423?ref_src=twsrc%5Etfw)
    

    • If there is a musical instrument that plays visuals, it would be great to have a “musical instrument that plays text” as well.

Current specific concept:

• Text displayed on the screen, with one character added per beat.

• Left alone, LLM will continuously generate text.

• Press buttons like playing drums to control the output.

  • Pressing the “a” button restricts the next character to the “a” row.
    • あかさたなはまやらわ
  • Pressing the “k” button restricts the next character to the “ka” row.
    • Imagining a sound like “ka.”

• Using this, press keys rhythmically as text is generated.

  • For example, pressing “a” every four beats can create a rhyming text in the “a” row.
  • Pressing “t,” “k,” “d,” etc., at the right timing can create a consonant beat.

• It would be desirable to control aspects like “output two characters per beat at this timing” or “insert a space at this timing.”

  • Essential for rhythm control.

• It should be possible to control and generate text with specific meanings and moods.

  • Adjust prompts for the base’s output strategy in real-time.
  • It would be fun to adjust parameters like “brightness,” “sadness,” “emptiness” with physical dials.

• Similar to VJ and DJ, it seems possible to perform improvisational poetry and lyrics continuously (blu3mo)(blu3mo)(blu3mo).

• A more concrete image emerged from [/frog96lab/notes on rhyming: non-vowelism, sense of rhyme, general rhyme](https://scrapbox.io/frog96lab/notes on rhyming: non-vowelism, sense of rhyme, general rhyme).

  • It would be fun to play with people like Frokuro if this is created (blu3mo)(blu3mo).

• Implementation:

  • The slowness of LLM is an issue.
    • Solution: Always prompt LLM to output future characters speculatively.
      • Generate characters in advance to cover all possible constraint patterns.
      • If there are n constraint combinations and generating k characters ahead, then n^k extra generations are needed.
      • Use it generously.

Other ideas:

• Sampling could be incorporated.
  • It would be exciting if suddenly familiar phrases appeared.
• Mixing:
  • Wasn’t mixing words something [ritar] did before?

For a rough prototype (blu3mo):

• For now, giving up on making coherent sentences and creating a sequence of random words?
  • Experiment with just sound to test if the desired outcome is interesting.

Let’s create an AI that can rap Part.1 - Searching for Rhymes - Dentsu Tech Blog Let’s create an AI that can rap Part.2 - Making ChatGPT Rap - Dentsu Tech Blog

• Useful references (blu3mo)(blu3mo)
• Rhymebetting made me laugh.

https://www.ipa.go.jp/jinzai/mitou/it/2022/ps6vr70000014g43-att/hyouka_tn-1.pdf

• System for the Rap AI project in the MITOU program.- In a rap generation method that considers rhymes, a backward generation approach is adopted. By using a Transformer Encoder-Decoder, rap lyrics with rhymes are generated by creating sentences from the reverse direction. Additionally, a verse generation system that considers answers was developed by creating training data that represents the relationship between answers and adding filters based on a unique evaluation function.

  • I see!
  • If it’s already decided to rhyme at the end, then this is great.
  • I wonder if it could be done more easily with a strong LLM in 2024 (blu3mo)

Might implement. §