chatbot1-9  Exercise 1: Splitting Long Replies
================================================

ex1.py (save in the project root):

  from bot.channels.telegram import split_message, utf16_len

  text = "\n\n".join(f"Paragraph {i}. " + "This sentence is here to take up space. " * 25 for i in range(1, 8))
  print("whole text:", len(text), "characters,", utf16_len(text), "UTF-16 units")
  for limit in (4096, 1000, 300):
      chunks = split_message(text, limit)
      print(f"limit {limit:>4}: {len(chunks)} chunks, sizes {[utf16_len(c) for c in chunks]}, largest {max(utf16_len(c) for c in chunks)}")
  chunks = split_message(text, 1000)
  print("first chunk ends with:", repr(chunks[0][-25:]))
  print("second chunk starts with:", repr(chunks[1][:25]))
  print("nothing lost:", " ".join(" ".join(chunks).split()) == " ".join(text.split()))
  emoji = "😀" * 2500
  pieces = split_message(emoji, 4096)
  print("2500 emoji:", utf16_len(emoji), "UTF-16 units ->", [utf16_len(p) for p in pieces], "| characters per piece:", [len(p) for p in pieces])

Run with:  py ex1.py

Output:

  whole text: 7103 characters, 7103 UTF-16 units
  limit 4096: 2 chunks, sizes [4057, 3042], largest 4057
  limit 1000: 8 chunks, sizes [997, 994, 999, 998, 993, 998, 997, 119], largest 999
  limit  300: 25 chunks, sizes [297, 299, 299, 294, 299, 299, 234, 297, 299, 299, 294, 299, 299, 234, 297, 299, 299, 294, 299, 299, 234, 297, 299, 299, 114], largest 299
  first chunk ends with: ' This sentence is here to'
  second chunk starts with: 'take up space. \n\nParagrap'
  nothing lost: True
  2500 emoji: 5000 UTF-16 units -> [4096, 904] | characters per piece: [2048, 452]

The emoji result: each emoji outside the basic multilingual plane takes two UTF-16 code units, so 2,500 emoji are 2,500 characters but 5,000 units. The splitter measures in units (a cautious choice: I do not know which unit Telegram counts in), so the first piece is 4,096 units, which is 2,048 emoji, and the rest is 904 units (452 emoji). Python strings count code points, so cutting at a character boundary never splits an emoji in half.

Where the first cut fell at limit 1000: after "This sentence is here to" (a space), not at a paragraph break, because the paragraph break sat just beyond 1,000 characters and the splitter only accepts a separator in the second half of the window.

WHY THIS WORKS AS AN ANSWER
---------------------------
The splitter finds the longest prefix that fits the limit, looks inside it for the last paragraph break, then line break, then space (accepting one only if it is far enough into the window to avoid tiny chunks), and cuts there; with no usable separator it cuts hard. Every chunk is at most the limit, the pieces joined back together contain all the original words, and each chunk is trimmed at the cut. What it does not do is respect sentence ends, formatting or
grapheme clusters (an emoji with a skin-tone modifier is several code points and could be cut apart in the worst case).