Using Machine Learning to Extract 3D Data-VBForums
Results 1 to 2 of 2

Thread: Using Machine Learning to Extract 3D Data

  1. #1

    Thread Starter
    Lively Member
    Join Date
    Dec 2016

    Using Machine Learning to Extract 3D Data

    Hello everyone,

    So I just started getting into machine learning and it痴 been a lot of fun. I took some online courses on Pluralsight to help me get started with TensorFlow, Googles machine learning library. However, even with the online courses I took I知 not completely sure how to handle the project I知 looking to undertake. I知 hoping there may be some people here who can point me in the right direction.

    I知 actually looking to train an AI to watch youtube videos of dogs and cats (yes cat videos). I want it to not only be able to identify the particular breed of animal (German shepherd, Collie etc..), but I want it to learn the general behavior of the animal. From what I understand, a convolution neural network would be ideal to use for identifying objects in images. However, for video is it any different?

    Now why would I want to do this? Well I want the AI to have a general knowledge of how a particular breed moves. Not so much the behavior, but more so how it moves and how it doesn稚. I知 trying to develop an automated solution for drawing and animating animals frame by frame.

    My thought is, if an AI, can understand the 3D proportions of any particular object or animal, and it痴 movements, then you should be able to direct the AI to literally draw the subject on screen in multiple frames. You could simply use photoshop for the canvas. I知 not looking to copy content. That痴 easy to do without training an AI. I want it to generalize it痴 knowledge and create original animations and designs based on what it has learned.

    So, I have been struggling to come up with a potential solution this. How much instruction does an AI need to carry out this task? I mean if I were programming this with a rules based approach, I would try and use a form of photogrammetry to produce a 3D point cloud based on video data, which could then be reconstructed into 3D in any fashion you like. This is essentially a form of computer vision. Sometimes when there is not enough 3D information extracted from a series of 2D images, interpolation is used to estimate where detail is to be added based on existing point data. There are a few libraries to handle this sort of operation. MatLab, MatplotLib (Python) and OpenCV .

    However, AIs do not use a rules based approach. If provided enough data (and the right kind), could it figure out how to get the desired output by itself? Or does it need a certain amount of instruction to get there i.e. photogrammetry?

    I suppose you could train a supervised learning network with certain 3D point cloud data so it knows what to look out for in the videos. Im not sure if I even need to use 3D Point clouds though? What do you guys think?

    Sorry for the long post...I tried to be brief but its rather complex.

  2. #2
    Join Date
    Jun 2013

    Re: Using Machine Learning to Extract 3D Data

    Before you jump into analyzing (and re-interpreting) existing movements (patterns which already "have evolved" over some time),
    I guess "doing it from the different direction" (from the ground up, "simulating evolution") also has its value (shouldn't be dismissed that fast)...

    If you choose a nice Physics-Engine (perhaps starting with a 2D-one, later switching to 3D)...
    and then define (according to certain "animal-types"):
    - "segments in the right proportions and with appropriate mass"
    - which are connected over "Joints" (which restrict certain movements, and allow others)
    - and also support "spring-like bindings between segments" (to simulate muscles and tendons)

    You can achieve surprising results even after only (relative) few generations of an evolutionary algo,
    in conjunction with a Neuronal-Network, all "set up for certain goals to achieve".

    Here's something like that, which demonstrates what one can accomplish with very simplified "Stick-figures",
    and a still comparably small Neuronal-Network (in conjunction with a simple 2D-Physics-Engine).
    (Youtube-Demos, based on VB6-implemented solutions by reexre, a forum-member here, might send him a PM to "talk shop"...):

    Learn to stay vertical:

    Learn to jump:

    And here a somewhat more complex stick-figure, that resembles a kangaroo

    I'm quite sure, that with a bit more time-investment into "better stick-models" (getting mass and segment-proportions right,
    as well as more realistic restrictions on interconnecting "Joints", Muscles and Tendons - accompanied by a larger Neuro-Network,
    you will get pretty realistic movements out of it, when you let it "evolve" over a few more generations than shown in the vids above.

    If you then managed to get good results in 2D, switch to 3D and repeat.
    If that works out as well, then you can "store 3D-Physics-Models" along with the "trained for certain typical Movements" NeuroNetwork-Params in a DB or something.

    Just my $0.02 (thinking about that topic, and your goals)...


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts


Click Here to Expand Forum to Full Width