Introduction to NTLK

Text is everywhere, approximately 80% of all data is estimated to be unstructured text/rich data (web pages, social networks, search queries, documents, …) and text data is growing fast, an estimated 2.5 Exabytes every day!

We have seen how to do some basic text processing in Python, now we introduce an open source framework for natural language processing that can further help to work with human languages: NLTK (Natural Language ToolKit).

Tokenise a text

Let’s see it firstly with a basic NLP task, the usual tokenisation (split a text into tokens or words).

You can follow along with a notebook in GitHub. Continue reading “Introduction to NTLK”

Advertisements

The overfitting problem and the bias vs. variance dilemma

We have seen what is linear regression, how to make models and algorithms for estimating the parameters of such models, how to measure the loss.

Now we see how to assess how well the considered method should perform in predicting new data, how to select amongst possible models to choose the best performing.

We will first explore the concept of training and test error, how they vary with model complexity and how they might be utilised to form a valid assessment of predictive performance. This leads directly to an important bias-variance tradeoff, which is fundamental to machine learning.

The concepts described in this post are key to all machine learning problems, well-beyond the regression setting. Continue reading “The overfitting problem and the bias vs. variance dilemma”

Effective military corps are Agile

I never read it but I have been told that the Marine’s Corp “Warfighting” manual  contains several similarities with how Agile projects and teams should be.
If you also don’t feel to read the dense 100+ pages document here is my shorter take, inspired from what I learnt during my mandatory military service time (well, I was not in the Marine’s but still in a rapid response force).

Embrace changes

Traditionally you associate military to huge command-and-control structures but that is not always the case.

Continue reading “Effective military corps are Agile”

NLP3o: A web app for simple text analysis

We have seen earlier an introduction of few NLP (Natural Language Processing) topics, specifically how to tokenise a text, remove the punctuation and its stop words (the most common words such as the conjunctions) and how to find the top used words.

Now we can see – as an example – how to put all these inside a simple web app.

The entire source code is available on GitHub, so you can better follow along; and you can see it in action in a Heroku dyno.

Screen Shot 2017-07-29 at 14.55.08 Continue reading “NLP3o: A web app for simple text analysis”

Logistic regression with Python statsmodels

We have seen an introduction of logistic regression with a simple example how to predict a student admission to university based on past exam results.
This was done using Python, the sigmoid function and the gradient descent. 

We can now see how to solve the same example using the statsmodels library, specifically the logit package, that is for logistic regression. The package contains an optimised and efficient algorithm to find the correct regression parameters.
You can follow along from the Python notebook on GitHub.

Continue reading “Logistic regression with Python statsmodels”

New Chinese Science Fiction

This time I do a digression from my usual themes to talk about Chinese Science Fiction. But you will see that is not a great deviation: some of the topics are familiar; as William Gibson famously said: The future is already here — it’s just not very evenly distributed.

I have always been a science fiction fan and I noticed in the last years that the genre is booming in China (maybe as a way to more freely talk about current affairs disguising them about speculative fiction?)

Here are three books that I enjoyed lately.  The links are to the authors or publishers pages, not any book retail page. Continue reading “New Chinese Science Fiction”