Plan - or not

Having a true Unsupervised Journey spirit, I have developed this bullet proof plan:

1. Try: Lets just do something, I heard OpenJourney works ok.

2. Feedback: See how it looks

3. Iterate: Start over with the extended knowledge

Step 1: Lets just try something

Google: “Use OpenJourney”, click on first non-advertisment link and try some prompts like

• “lionel messi”

• “a caricature drawing of lionel messi”

• IF I would continue for two more hours, I could get the third picture. But I didn’t.

TLDR - it was somewhat promising - but I wanted to get closer to my fathers mastery:

The difference to the generated pictures is obvious. For example you can see there are many intricacies to be perfected, especially:

• a more nuanced skin,

• a stroke-detailed hair and

• some funny object for the character (this is Ivan Miklos, a former finance minister of Slovakia).

Step 1: Take the idea of custom avatars

Back in spring, everyone was hyped about their AI-generated avatars. It is pretty cool - you upload 10-20 pictures of yourself and you could generate (almost) whatever pose you wanted. For example you being on a moon riding an unicorn or your next LinkedIn headshot.

How this works under the hood? It’s essentially like teaching an adult artist on new knowledge. You repeatedly show the model new pictures of the new object (the person), and then you ask the model to draw it. On the neural network layer it essentially further trains the general model with a lower learning rate. This process is called fine-tuning, and that new knowledge can also be a new style.

Training the style

So now I have three problems:

• Get the training data to teach the knowledge of dads mastery

• Fine-tune OpenJourney

• Generate outputs (and iterate on the above).

For the first, it was kinda easy cause my dad gave a deck of cards in an uniform style of famous people in czechoslovakia. So I just went ahead, snapped pictures on my smartphone and voila!

For the other two, luckily there are quite a few services doing model fine tuning for me. So I have tried LeapAI, LeonardoAI and RunwayAI - and after early results I have decided to commit to LeapAI.

The process was roughly:

• Hit Create New Model

• Upload your sample images - 20 is enough

  • Wait for model finishing fine-tuning - that took about 30mins.

• Go to Playground and enter your prompt.

PRO TIP: Here, you should pick a well-known well-defined recent subject. Cause I first picked Albert Einstein first (who has no high-quality photo) and then Cristiano Ronaldo (and there are many very famous Ronaldo’s).

Conclusion from the prototype

I was pumped that it looked pretty good for the first pass so I shared it with my brother and he said “wow”.

But it was still far from the desire, and first the first Iteration I had a few ideas to improve:

Input images

• Use the good old scanner instead of my camera for consistent image quality

• Crop pictures to 512x512, with faces only with cropped out backgrounds

• Used 50 images instead of 20

Prompt engineering

• I have used ChatGPT to iterate on prompts. This ended up being more of a black magic than a scientific process as I ended up trying almost 100.

• Inference parameters experimentation:

  • Steps: (more seemed to work better)

  • Temperature / Prompt strength (more variance in output)

  • #Images (more output, higher chances of success :D)

Next Iteration: Consistent Data

My intent was to train portrait drawings, so I focused the input data to do that. These images were:

• scanned at the same resolution, placement and cropping style (previously just camera pics).

• manually post-processed (removed background, unnecessary parts, …)

• scaled to 512x512 pixels which is the same resolution as the base models are trained on

Lets now compare how much the underlying data helped to achieve our goals. We are using the same prompt “a photo of lionel messi” with the difference of our fine-tuned model - this was a STEP CHANGE. I got pumped.

Next Iteration: Better prompts

Now, I best-effort tried to describe what I want to achieve. Most of the work was finding the right words for “caricature” (was “a caricature drawing of”) and my dads style (was “crosshatching”). The final prompts:

Positive: a caricature drawing of {person}, 
headshot, profile, slightly smiling, lines, intricate hair, 
fits in frame, crosshatching, textured, fine strokes
Negative: duplicate, malformed, color, mutated, jpeg artifacts,
out of frame, digital, smooth

Next Iteration: Parameter Tuning

TLDR; I just pretty much tried all parameter combination using a “binary search” process:

• High-level best params

  • Prompt strength (temperature): 4, 7, 10, 13, 16

  • Steps: 20, 30, 50, 70, 100 (NOTE: more steps takes longer)

• Narrow it down for the best combo - say it’s 10 strength and 70 steps then I would try:

  • 8, 9, 10, 11, 12 strengt

  • 60, 65, 70, 75, 80 steps

PRO TIP: Set the Seed parameter during experimentation to get deterministic and more comparable results. Also use as many output images as possible.

Oh, and wait what did my dad say? 🤪

Well, how should I tell my dad that machines are after his job which he is super-proud of?

So I ended up ordering him a canvas print of the 32 soccer captains from last FIFA world cup and patiently waited for his call 😀

TLDR on response: Denial.

So after text and images, I went onto exploring voice AI.

The mainstream solution is to develop a personalized “ritual dance” which helps us to get started. To start writing we usually follow these obvious steps:

• First, we do the dishes.

• Next, have a nice long shower and dress up fresh.

• Finally, we admit we gotta get out of the house. Obvious choice is to get a coffee - or go to our “coffice”.

Still, even though there is nothing else to do, we still just stare at that large blank textbox.

Solution

What if I told you there is a better way. Let me explain by sharing my own journey.

Journey to the solution

Some Friday in California - It’s the last day to fill in the corporate grunt work we all fear. Writing up yearly Performance Reviews (or “Perf”).

Scary or not, you have to write one for yourself and one for all your peers. And even though I blurrily know what to fill in the questionnaire - I just get overwhelmed by trying to fit the past half year into my brain.

So here I am myself, starring at that blank form after multiple visits to the microkitchen. Still hesitating to start - I go for advice to my buddy ChatGPT.

That’s when I notice the solution is right in front of me - the voice input button for ChatGPT. Yes! I can just talk my brain out in 10 minutes and it will fill it for me! So I booked a meeting room and I literally dumped all those projects, achievements and learnings and told it to output the desired structure.

Hitting the stop button already yields a huge relief - all I wanted to say is already out of my brain! It was like crossing the Bay Bridge on rush hours after a long long wait.

The output draft itself needed editing - but every content person knows that editing an existing text is much simple. So just a bronze bullet - not a silver bullet solution^TM.

Other applications

It’s not just Perf - there are many repeating use-cases I use the super-long-voice-to-GPT button:

• Discuss engineering designs while watching kids at playground [example]

• Weekly sprint task planning [example]

• Writing draft emails, like formal ones to .gov

• Daily journalling

• Researching solutions [example]

• AND admittedly, to start off this blog-post 🙈. In this case the draft was so bad I rewrote the entirety of it - I pinky promise.

PLUS, you can do this while walking to your coffice, or driving home.

Limitations

Some I have experienced:

• Bad internet and loss of your long voice input

• Editing options

• Searching through past notes

Appendix: Previous Limitations of Existing Tools

Voice assistants

Every tech-savvy individual knows about the voice assistants like Siri and Google Assistant. Their primary functionality lies in their ability to execute short commands, whether it's setting a timer or jotting down a calendar reminder.

For long-form content, historically, the solution was to lean on a real human - in a role of an executive assistant. Think of those high-ranking professionals dictating summaries of their high-profile meetings. There was clearly a gap between these two extremes.

The voice-to-GPT solution lies in between these.

Apple / Google keyboard

Me and a few folks tried it, but the transcription wasn’t there quality wise. They will clearly be better, after which you can just use that.

Humans create myths - and in software engineering we have the mythical concept of a “10x Engineer”. It simply defines as a

“developer 10x more productive than most other competent developers”. 

For this blog, the key takeaway is the feeling of having superpowers. When I started to pair-programing with my team of 10 opened chrome tabs of GPT experts - really felt like a 10x engineer.

Why this feeling? Cause you feel like you can fly - which is even better than actually flying. You just know there is no integration abyss you cannot overcome, that even the most mundane data translations would be a swift layover, or there ain’t a stack trace deep enough you can’t climb (with the 100K token models especially). Just knowing this for a fact gets you so much dopamine. No more of getting stuck and of that “I am gonna motivate myself in the microkitchen” stuff.

WOW

This blogpost is the story of how that happened for me, sharing my usual prototyping workflow with type of prompts I do for insights.

Prosaic Intro which you can skip

This engineering project had an unlikely start - a long-weekend coinciding with my spouses birthday:

• She: For my wish, you know Peter, you a pretty good engineer …

• Me: [#homerdisappear]

• She: … and there is this thing I keep mentioning …

• Me: [#intensifies]

• She: … you remember when I was doing that annoying meeting note-taking work and you made this magic tool to transcribe it through GCP some years ago?

• Me: Yep, it didn’t quite work.

• She: Well, I heard the tools got better, while my problem is also more painful. I go to many events returning with my brain buzzing with all the conversations. And you know how much I care about relationships - so I obviously want to follow up next-day. But only if I would remember what we talked about.

• Me: Hmm, maybe we can automate that. What would you like?

• She: I would like to have something like the partners at my firm - an assistant they can just call and do the information organization for them. Maybe I record something and that ChatGPT summarize it.

Lets get started

First, lets just try what’s out there for our task:

• Input: A voice memo recording as mp4

• Output: Excel row for each person with contact notes as columns

I like to get my hands dirty right away - to better understand the problem. Having not much expertise Google suggests using OpenAI Whisper for transcription, so I check their API, which looks modern, therefore I just write a short script and call it (Note: I tried to generate it but ChatGPT from 2021 didn’t know the ChatGPT API from 2023).

Running the script yielded “fileformat not supported”. WTF OpenAI, your docs say you support .mp4. I guess you move so fast that your docs are already outdated! Luckily, I can just use ffmpeg to convert to mp3. Now it indeed works pretty well!

Next I take that output and prompt: “Given the transcript, generate these 10 columns defined as follow” - slowly the response reveals what I was looking for and I am feeling like done:

• Me: “Hey honey, look here is your present”.

• She: Oh nice! How can I do it myself?

• Me: Oh you can just send it to your computer, run this python script, you get a csv, and in your excel you just import special → add row and you done. Dead simple.

• She: [not a happy face] - I mean this can be useful for other people too.

One face is thousand words - so clearly I got to automate this.

Design work [technical]

TBH it’s been a LONG time since I built something ground up. I was thinking of having a simple domain, and something in AWS to run ffmpeg and GPT queries. I felt stuck - but luckily, I had my local expert at hand to figure it out alongside my good DevOps guru friend in town:

• Me: I need to build a system where users send an MP4, and they get an Excel file back via email.

• ChatGPT: Have you considered direct file uploads for a better user experience?

• Me: Developer ease is key for me, I don't want to host any servers.

• [cut for brevity, see appendix]

Design Review

They alerted me that ChatGPT is a like a highschooler - so lets spend some effort reviewing the resulting design.

I reviewed our requirements:

1. recording voice [up to 20 minutes]

2. run transcription [up to a minute]

3. run customized gpt prompt [up to a minute]

4. send the structured output

5. served in cloud

Recognizing that from these, only part 3 is our “core custom business logic”. Ideally, part 3 is the only part we need to develop ourself. The other parts of 1, 2, 4, 5 can be hopefully re-used from the wild.

The design would be then:

• having Apple voice memos to record

• will get that to cloud via email w/ attachment → lambda → email

• the 2, 3, 4 logic would be served out of that lambda

What I liked:

• it was really minimalistic,

• plus the entire thing can be processed asynchronously (with retries)

• without any need of frontend or a server.

• AND lastly easy to explain: it’s like sending your secretary a voicemail.

Implementation - The birth of a 10x Engineer

<history-time> Back to my elementary school times. I still remember the thrill when I was learning to code - it was kinda like discovering legos on steroids - as you could build out your ideas quite well.

After my university, with me entering the workforce, there was always the tradeoff between my childish builder passion and our business needs. Slowly, coding become more of “work which needs to get done”. </history-time>

At present time - prototyping with AI agents is like coding on steroids again. Or how my friend said it: “it’s just much closer to speed of thought”.

For the prototype implementation itself, I would say 85% of the coding job was automated. The job was split so that AI agents wrote most of the logic - while I brought in the intent, supervision and maybe most importantly clicking the right buttons. See Appendix for details.

Conclusion

Great job us - we got what we wanted!

But the true learning here is from the process we went through, and not he solution we created. Here we become trailblazers and not just mere hikers following a path. We got dropped at a random spot in the wild, and using magical tools we made our own path. Going through this Unsupervised Journey we gained confidence so we can blazer our paths repeatedly.

So I encourage you too, go embark on a side project for your dear, and learn to fly with those AI agents.

== THE END ==

Appendix 1: Rest of the design discussion

• [repeated] Me: Developer ease is key for me, I don't want to host any servers.

• ChatGPT: You could use SendGrid to process incoming emails and trigger a webhook. Let me break down the steps.

• Me: Tell me more about the email parsing.

• ChatGPT: SendGrid has an inbound parse feature. I can also suggest alternatives.

• Me: Hm, that's too much novelty for me to implement. Can we do the entire system in AWS?

• ChatGPT: You can set up the services like S3, SES, Lambda, and SQS, and I outlined the procedure.

• Me: Instead of Excel, how about a Google Sheet?

• ChatGPT: Use the Google Sheets API from Lambda. It allows creation, populating, and sharing.

• Me: Can I get a sample Python code for that?

• ChatGPT: Sure, I've provided an outline for authenticating, creating sheets, populating from CSV, and sharing with users.

• Me: Please summarize the discussed design in bullet points ordered by the user experience.

Appendix 2: List of AI agent Chat Threads

• Setup a new domain (should have used Namecheap)

• Emails to katka.ai redirected

• Convert Audio to MP3

• SES Email Setup on AWS

• Email Attachment from SES

• Parse SES Email Attachments

• Lambda function which can handle ffmpeg and Python code

• CSV to XLS Conversion [stretch, didn’t happen]

• Excel with Multiple Sheets [stretch, didn’t happen]

• Convert Python objects to JSON

• HTML code for this table

• Sassy Follow-up style

• Parsing Email Timestamps

• Why this error in my lambda script? [a LONG thread indeed]

• Email Alert on Lambda Failure.

• Extract persons in the transcript

1. Why another blog in 2023?

In an era with an average attention span of just 10 seconds, you might be tempted to close this tab (quick hint: Cmd+W). If you're still here, the lure of attention-economy might suggest I dangle a few enticing tidbits to ensure you continue reading. If you've made it to the end of this paragraph, then my in-depth content sparkled with some humor might be right for you.

Now, for the core reason of (re)starting my blogging output. The key input fact is that I am old enough to admit that my best way of my communication output is trough writing (code). This become apparent during my recent chat with my dad:

• Dad: “I just had a refreshing beer on a hike - ask me anything.”

• Me: “So, what about me pisses you off the most?”

• Dad: “Peter, I very often find it hard to decipher your words”.

Thanks dad. I tried to keep a positive mindset that I contribute a tough training dataset for those voice-to-text AI models for the benefit our humanity.

Anyway, so blog it is.

2. What it will be about?

The above reads like a narcissistic desire to be heard - lets judge that later.

For content, my primary goal is to share stories from what is now an exploratory phase of my professional life. Ideally, it’s a win-win for us, you get insights from my missteps, while making those public holds me accountable to navigate thoughtfully.

For style, I decided to call this blog Unsupervised Journey. A cross between my unsupervised learning and passion for mountains. The name should voice my life-long belief that the best way to explore uncharted territory is through combat. Yes, you will short-term spend a lot of sweat on your inevitable mistakes - hopefully traded off for long-term understanding and confidence.

3. Post cadence

Enough talking, lets get to something graspable. Until end of 2023, my aim is to roll out at least one blog piece weekly. To give you a taste, this is on the map of my blogging journey right now:

• Overcoming my AI skepticism OR the night my wife asked: So is your job obsolete?

• Artists denial: Automating My Dad's Caricature Sketches.

• Friends raising concerns over my relationship with ChatGPT.

• Frontend looks easy OR how hard it can be to create a voice-record button.

• AWS SAML - great promise OR how my daughters learned to swear.

See you on the road!

Subscribe now