How me ? – Experience Training Students at PA College of Engg.Mangalore, with Python.

One fine Wednesday evening one of my friends Sarath, whom I met only a week before,called me. Though was working late, I was relaxed as a bigger chunk of code I wanted to deliver was delivered and was happy with the outcome. Basically that means, I was all in a very very good mood.  


But Sarath was not in the same mood. He started the conversation saying, Sumit, this is a distress call. I was like, why mate, what happened ? Then he briefed his position to me.


Sarath is a very motivated and socially responsible guy who takes huge efforts to push the Free Software Movement of Karnataka forward and FSMK had offered to train students around Mangalore Engineering Colleges which was gathered under PACE‘s initiative to bring open source(GLUG Mangalore) to the kids.Towards that final moment, when there were only two days for the event to happen, the Guys who had offered to go with Sarath to the session backed off. His need was to fill the positions.


I being a rails guys for really long and my knowledge of Python dated even before that, obviously would have had to think twice. But donno what, I just said, Dude! that is it! I’ll go to Mangalore.

It was for the first time I was going to see those guys who were going to be my fellow colleagues for the rest two days and there were as usual thoughts, how they would be.The organisation committee had put 1-2 years extra in the age field for everyone(in the bus ticket they sent me). So I was under an impression to see some really “experienced” guys up there.


Well for my luck, all were fantastic guys!


I met Chandrashekar, who works as a Team lead, at ZeOmega, Emil and Mayank who works with SourceBits. They were equally cool when they discuss technology and other stuffs. I got to know a lot of new stuffs from there. And to my surprise, Chandrashekhar was working on ETL (Extract Transform Load) projects for a while now.It is important for me, because the project that has took away my sleep and movies from my life for the past 1+ month was ETL. He helped me with a lot of his experienced opinions and new techniques to handle stuffs. Have to get more from him. Plan was to do it on the Day 1 evening, but that got switched for a movie plan later 😛


Chat with Chandrashekar was more on the topic Philosophy than on Technology through the travel. He just has got a lot of experience thinking on philosophy. I being pretty open for discussions on philosophy, just loved it!


Day 1, morning, we were picked up from the bus stand and was dropped at PACE’s guest house. Have to mention, the food was awesome. And that was not the thing that surprised, us. The food was the same that they served at their hostel messes, and we started feeling jealous for the kids here 😀


Had a power nap and we were ready for the day. Not knowing what and how the scene would be there in the college. We were introduced to the HOD of PACE Mr. Ganesh. Such a phenomenal guy. He is an asset for them. The students here are all words, when they talked about their HOD. He deserved that, of course..

 And rest of the story is at here(day 1) and here(day 2)

Day 1 of 2 days with my py-kids

Day started with first intro by Chandrashekar. He had created the correct slides, and gave a perfect 101. It was sad that guys and gals where not even ready for that. FSMK, GLUG and all FOSS orbs out here has to do a serious work on this.they need to know what open source and how cool is it.
First lab session, and I was assigned in Lab D. So sleepy and inactive guys and gals to deal with.
     But, I’m happy the Gender ratio was equal and better on the girls side. This is something the Open Source community has yet to improve on. I was trying to follow the questions Chandru had set already. And I couldn’t feel that the kids where gaining anything, because the time frame we allotted them was not enough for them to catch up.
After asking questions and getting all eyes staring at me, i started saying, guys, responses please. That is what would take you into height you have to fly. Respond when you know something. Respond, and say, when you , don’t know something. Which essentially means, respond, no matter what.
After lunch second session by Chandru was on read and write – Files  and Classes. And there where four questions for the kids to work on. And right after his talk when people was still not responding, I thought it is time for me to step in and give them a talk about how they should see themselves if they have to rock in this industry.
It was time for me to be in action – My Lightning talk
I was not so happy with the 0 response principle these kids carried. I wanted them to wake up and stand for themselves. Forget about the company taking you in and training you and feeding you. I wanted them to understand the need of working for themselves.
After giving a quick talk to the all four teams it was time to be back back at the Lab session.
This time I did not start with the coding part. Instead I gave a 10 mins talk, a motivational talk. On how the job would be. How hard it would be to survive. How you should struggle. And more importantly how to get into the best companies out there, rather that waiting for MNC’s to come and pick them in, and put them into the companies choice of technology – which mostly would be the legendary manual testing.
I tried to achieve the ‘open source and how cool is it’ part here. Hopefully some must have grabbed something, if not all things, and they never had any hands on experience with this. Or may be was not aware that such a cool community exists.
I was under a tight schedule under my work, and when I accepted this offer to take session. All I wanted was to deliver some value to the community I belong to. The community which pitched me into a wonderful job. And is still caring for me by making me enthusiastic with its own mind blowing developments.
I told them, all I wanted to take away from here. All I can take away from here is just, satisfaction. Satisfaction, of teaching some students how to set their career in a beautiful and promising track.
Well  that went well, and I got really got responses from kids – that was unexpected.
It was time to start the lab, and I had a plan this time in my mind. I did not want to traverse the questions which they where asked to solve. Instead, I wanted them to understand the concepts. I collected the the 4 questions together and made it into a single question and started working.
The first two questions was file read and file write. The third was Class  and the forth was Dictionary.
I planned it like, create a Dict. That was done, and people pretty well started understanding how to set the key value pair and how to add/append data into it. Then it was creating a class to hold those data. Made them pass those data into the class and make an object of it. For this second short 1 hour session which also included a motivational talk, this was enough.
Had been asking them if they understand or not. If they do not answer when I ask ‘did everyone understand ?’ then the question gets changed ‘ who all did not understand  ? ‘
including both sessions, after 2.5 hours of effort, people starred responding well. At least they daed to say, they did not understand some concepts, which was valid. Coz, I even had to face somebody who did not understand and array and relations ship between index and value in it. Which is totally fine. After all, I’m there to explain ( though supposed to be explaining PY that data structure concepts).
I gathered all around my laptop and showed then step by step. Asked them questions. And stressed until they decided to give answer.So here is the thing, If they don’t know answer, they don’t talk. They don’t even give a try, or a guess at least. All I would have asked would be a yes or no question, and a guess is pretty easy there. But I was patient enough to wait until till decided to give me the answer. though it was wrong, I don’t care. I know they are fresh heads and they are not expected to be correct all the time. But for me they are future engineers under constructions, and have passed 3 years under the build process called Engg. classes. I expect them to be responsive, and alive.
BUt towards the end, either they developed a confidence within them, or developed a trust in me, that, they started to respond pretty well. They started participating. And when there is a two way participation happening there is nothing better a session would need of being effective.
I’m hoping I did justice to them on Day 1.
Well it was time for some evening masthy (fun) and we will a cool college guy went for a movie, prisoners. And luckily that was a fantastic movie. We had a wonderful time. And best part was, we never felt we just met each other last night. All were as if we were friends from the college, and rather, we were from college! It was a very pleasant and for some reason nostalgic night.

 Day 2 overwhelming…is coming…here

Day 2 of 2 day with my py-kids (Experience Training Students at PA College of Engg.Mangalore, with Python.)

Today is day 2, which is complete 3.5 hours lab session. I am preparing for the day. I have travelled all the way to contribute. To the new guys who are yet to step in to the industry. All I want to take back is a satisfaction that I made some people learn. And I believe I’ll do that happen today.


I was hoping the guys and gals will be pretty energetic today.But what happened was not what I expected to get. They were the same. Not at all responding. I had to spend 20 mins talking to them, giving enough gaps, that the class will not proceed without them responding. Ultimately, when I saw that I was losing and so I tried a different approach. I wanted them to see me as a friend rather than a trainer. They were still in the school college stage where the guy standing against them across the other end of class room is a Sir or a Teacher. After repeatedly asking them to call me by name over ‘Sir’.


‘saar’… again, I couldn’t get many of them to forget this word…


But having said all these, all the guys and gals there where lovely… Given a chance again, I would take classes for the same batch any number of times…


I couldn’t find a face that was there yesterday btw. Though there must be something wrong with my class that I lost a student 😛

But later found out that since a parallel batch had a drop in students,my lab had issues with internet.


After calling all of them close to me, I started showing my laptop screen to them and a picture of mine sleeping on the beanbag at the office. That was one step that brought them close to me. They almost started feeling free. Well that was one correct step, as some feedback later included this instance being promoted by them. What I was trying to tell them was, if you have to get this freedom at work, you need to make yourself worth and for that you will have to struggle. And this is where it all begins, start getting involved, and start it with responding. Today I’m asking you people. Tomorrow, when you might be in front of me where I’m interviewing you, I wouldn’t even ask you anything and you might have to love yourself all by your words. So start talking. Learn to express yourself.


Then slowly started showing them small code snippets running in my console. Occasionally triggered my irb(interactive ruby shell) and showed them the python equivalent in ruby. Well the real need arouse, as I’m very much used to ruby that I was typing everything the ruby way in py shell, and was failing at times. And thus had to show them, that’s how it is in ruby.

But that very well helped them understand language as a medium, and I hope that would have set their mind understand scripting language principles.


Asked them questions, why this is like this and that is not like that kinda questions.

What is the type of dir(‘str’)

whats the type of dir(str)


and so on…until people started giving me correct answers. I was feeling like they are getting the concepts correct by this.


Once I started feeling like they are on right track, I asked them to group themselves into four teams, without looking which college they are from. They all gathered that way and now it was time to get the real game running.


Motivational talk, Career guidance was among topics I concentrated on, unknowingly. Python was an added advantage for them. Thus says their feedback.They are pretty happy about that.

I was lucky enough to be assigned with the most crowded lab, as the lab I got was the most spacious one. Over all, in every means, I was having a fantastic time there. More than teaching, I had learnings. I learnt to throw hesitance away and have definitely lot more.



Thanks to Anubha Upadhya M for all her help proof reading and making this post cute as it is now!

My Project’s Take Away from PyCon’s Pandas session by S.Anand

I being a regular rails guy since last 5 years, my appearance at the Pandas session was exclusively a project related one. And I already had hear high about the speaker of the session S.Anand of Gramener. My knowledge with python was of working for 6 months, that too around 5 and a half years back. So, I consider myself as a beginner when it comes to python.
The Project that I was working with had huge Data ETA (Extract Transform and Load) running on ~350 databases, with ~40 tables in each. The average rows count of visits table alone accounting to nearly 2 million rows each in each database. My project deals with moving all these data into one single database. Lots of number crunching has already gone into the population of visits table, and the data in some of the tables dates back to 5-6 years. The new implementation of visits are so complex that the number crunching pattern itself has been improved. So huge Transformation is manifested.
Above everything, we are running on a tight ETA, and very short window of down time for the migration.
My team had already put everything aside and started the work. But, I wanted to explore if we are missing any better implementation by any means. And that brought me to pyCon.
Talking about a minor complicated scenario. Where earlier we had a single table, now we have divided the columns in the same table into two different tables with a has one relationship between them. And the rows in the table are moved on to three different tables. It is evident that there will be various changes that was introduced to depict this big difference in the ER(Entity relationship).
(OLD) TableABCDetails 
TableA \
TableB – DetailsHasOneTable
TableC /
What makes Panda a Perfect fit for our need ?
Panda has got read/write(IO) facility for CSV, JSON, EXCEL, HDF, STATA, HTML, CLIPBOARD, PICKLE and our own SQL.
The session though was using CSV as the data source. The fantastic ORM and DSL provided by Pandas, will only want us to know a single difference
which is,
read_csv(‘…’) vs read_sql(‘…’)
to_csv(‘…’) vs to_sql(‘…’)
Updating a new Column with data, or adding a new column with data derived through an equation is a single step. No iteration involved, at least in the developers code. These all are again going to go into mysql as sql bulk create.
Complex calculations turns into a simple one-liner this way.
…a lot more of it’s feature will be covered in my later posts…
Get the database dump, and load it to Pandas, Do the Transformation, and them export as Sql. And source the Sql output from inside the target database. tada.
What all are the technical gains ?

Py has got a fantastic GC(garbage collector) compared to ruby, If I’m not wrong.

Py execution is fast compared to ruby.
Directly doing a bulk update on sql, than updating by iteration under for…loop in ruby using Active::model create method is much faster. And consumes very less memory. Will provide a Bench mark later in my next post.
The overhead of having huge data in memory is also less when run by pandas over ruby on rails. As Anand mentioned, pandas is the fastest and efficient way of dealing with Big-data with python, and probably with many other languages. Any way, my experience at the work shop confirms, that it is brilliantly fast.
In terms of code development time,
 – since no for…loops and similar interpolations are done the traditional way, that saves the dev a lot of time while coding the data transformation part.
In terms of process execution time,
 – The absence of ruby For…loop itself adds efficiency, and python has got better underlying C implementation.
 – Above all of that, In pandas, a lot of thoughts by data scientists and mathematics evangelists have been put in, making it cutting edge at what it does. It exclusively does handle big-data manipulation. It is born to maneuver custom code developed by a regular web/software developer.
…Currently our ETA doesn’t support such a huge change in technology usage, but I’m so impressed(and resting high hopes) with this that I will definitely make time to re-implement my project with Pandas/Python and probably that might get Acceptance too, seeing the whole lot of advantages this method holds.
All of those will be another post for sure…
A big thanks to Anand for his patience, brilliant presentation and technical skills. His knowledge is vast, and his humbleness is deep. I’m impressed by your talk.