Prerequisites - https://news.ycombinator.com/item?id=12719874[Calculus - ycomb] - https://mml-book.github.io[Mathematics for Machine Learning] - http://cs229.stanford.edu/section/cs229-linalg.pdf[Linear algebra review and reference by kolter and do] - https://probabilitybook.net[Introduction to probability by Blitzstein and Hwang] - http://cs229.stanford.edu/section/cs229-prob.pdf[Review of probability theory by maleki and do] - http://faculty.marshall.usc.edu/gareth-james/ISL/[Introduction to Statistical Learning] - https://ml-cheatsheet.readthedocs.io/en/latest/ Machine Learning - https://www.coursera.org/learn/machine-learning#syllabus[Machine Learning coursea (andrew ng)] - https://scikit-learn.org/stable/tutorial[scikit-learn Tutorials] - http://aima.cs.berkeley.edu/[Artificial Intelligence: A Modern Approach] Other - https://www.geckoboard.com/learn/data-literacy[Improve Your Data Literacy Skills and Make the Most of Data] - http://amid.fish/reproducing-deep-rl[Lessons Learned Reproducing a Deep Reinforcement Learning Paper] Python - An Introduction to Statistical Learning https://www.statlearning.com/ - pycon-pandas-tutorial https://github.com/brandon-rhodes/pycon-pandas-tutorial - PythonDataScienceHandbook https://github.com/jakevdp/PythonDataScienceHandbook - Python for Data Analysis, 3E https://wesmckinney.com/book/ ---- Building LLMs from the Ground Up: A 3-Hour Coding Workshop https://news.ycombinator.com/item?id=41412256 Taming LLMs - A Practical Guide to LLM Pitfalls with Open Source Software https://www.souzatharsis.com/tamingLLMs/markdown/toc.html ---- - https://learnaifromscratch.github.io/ai.html ---- Todo: big data links - / ## Log ``` 2023-12-18: Bear in mind that ML skillset is now bifurcating into two components. On the one side are the people who work at places like OpenAI/DeepMind/Mistral/etc, who have billion dollar compute budgets. They are the ones who will create the foundational models. At this point a lot of this work is very technically narrow, dealing with CUDA, GPU issues, numerical stability, etc. On the other side are people who are using the models through the APIs in various ways. This is much more open-ended and potentially creative, but you don't need to know how QLearning works to do this. It's a bit analogous to the situation with microprocessors. There is a ton of deep technical knowledge about how chips work, but most of this knowledge isn't critical for mainstream programming. ``` ## ML For most software engineers looking to use ML, starting with existing frameworks and knowledge and drilling down as necessary is the most prudent method. Abstraction helps you to be productive. It's certainly good to understand everything all the way down to the actual physics behind computing, but its not necessary especially at the start. I don't have to know exactly how logic gates work to program in JS and make a difference. I assume the same applies to ML. Motivation is what prevents most people from learning difficult to learn skills. Finding ways to produce value early on can help keep you motivated. Having good fundamentals in the maths will help you pick up new methods much faster as they pop up. And especially if you want to come up with new methods (as in research). Ultimately to get a job and make money in the area, you need to solve customer problems, starting with libraries and fine tuning is what needs to happen first. **some learning ideas** - https://spinningup.openai.com/en/latest/user/introduction.html#what-this-is - https://github.com/jacobhilton/deep_learning_curriculum - do the fast.ai course (https://www.fast.ai/) - https://rll.berkeley.edu/deeprlcoursesp17/ - stanford cs231n on vision basics(https://cs231n.github.io/) - cs234 language models (https://stanford-cs324.github.io/winter2022/) - https://fleuret.org/francois/lbdl.html - https://karpathy.ai/zero-to-hero.html - https://course.fast.ai/ - https://colah.github.io/posts/2015-09-Visual-Information/ - https://agentydragon.com/posts/2023-01-11-how-i-got-to-openai.html - implement a paper **LLM links** - https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/ - https://www.youtube.com/watch?v=zjkBMFhNj_g - https://simonwillison.net/2023/Aug/3/weird-world-of-llms/ - https://magazine.sebastianraschka.com/p/understanding-large-language-models - https://drive.google.com/file/d/1BU5bV3X5w65DwSMapKcsr0ZvrMRU_Nbi/edit - https://cyc.com/ - https://finbarrtimbers.substack.com/p/five-years-of-progress-in-gpts - https://kipp.ly/transformer-taxonomy/ - https://spinningup.openai.com/en/latest/ - http://d2l.ai/ - https://arxiv.org/abs/1706.03762 - https://arxiv.org/pdf/1810.04805.pdf - https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf - https://www.alignmentforum.org/library - https://news.ycombinator.com/item?id=34474043 - http://nlpprogress.com/english/intent_detection_slot_filling.html - https://github.com/google/sentencepiece - https://www.youtube.com/watch?v=9uw3F6rndnA - https://finbarr.ca/prompt-engineering - https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b - https://colab.research.google.com/drive/1KDSJKhZDd5fdbnLTalPKcjS_IDu0Q968 - https://dfdazac.github.io/06-neural-networks-numpy.html - https://finbarr.ca/5_years_of_GPTs.pdf - https://finbarr.ca/five-years-of-gpt-progress/ - https://www.arxiv-vanity.com/papers/2310.17680/ - https://www.latent.space/p/ai-engineer (for fake persona, name it one of first AI engineers) - https://docs.google.com/spreadsheets/d/1C9BSXdwXDpmT-FbSvp2rCIgPTuM5ef0y9Y0KtjeCOF8/edit#gid=0 - https://roadmap.sh/ai-data-scientist - https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ - https://bbycroft.net/llm